Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation

Knowledge Distillation (KD) is a widely-used technology to inherit information from cumbersome teacher models compact student models, consequently realizing model compression and acceleration. Compared with image classification, object detection more complex task, designing specific KD methods for non-trivial. In this work, we elaborately study the behaviour difference between obtain two intriguing observations: First, rank their detected candidate boxes quite differently, which results in precision discrepancy. Second, there considerable gap feature response differences prediction student, indicating that equally imitating all maps of sub-optimal choice improving student's accuracy. Based on observations, propose Rank Mimicking (RM) Prediction-guided Feature Imitation (PFI) distilling one-stage detectors, respectively. RM takes teachers as new form knowledge distill, consistently outperforms traditional soft label distillation. PFI attempts correlate differences, making imitation directly help improve On MS COCO PASCAL VOC benchmarks, extensive experiments are conducted various detectors different backbones validate effectiveness our method. Specifically, RetinaNet ResNet50 achieves 40.4% mAP COCO, 3.5% higher than its baseline, also previous methods.